PHP MySQL - Import million of CSV records into table in short time Using LOAD DATA
14-04-2017
Using LOAD DATA sql query, we can import large csv file into database table
I've always been trouble with uploading large CSV files into SQL databases. When we upload a CSV file, what we do mostly is, read each CSV records one by one, and we run an insert query inside a while loop. this is ok if the CSV records are in thousands. When we upload large CSV files using open and reading each record, this takes a huge load time. The solution is, we can use SQL LOAD DATA query. this will direct upload CSV records into the table.
Before starting, here is some configuration that needs to be done in php.ini
Edit post_max_size and upload_max_filesize in php.ini , it should be greater than size of CSV file being upload.Uncomment mysqli.allow_local_infile = On in php.ini, So that, we can read data from upload CSV File.
That's all configuration we need to do. Next, I'm gonna show the code.
<form method="post" action="" enctype="multipart/form-data" class="card p-4 demo-form">
<input type="file" name="file" required class="form-control m-2" />
<input type="submit" name="importSubmit" value="Import" class="btn btn-primary btn-lg m-2 " />
</form>
<?php
if(isset($_POST['importSubmit'])){
$csvMimes = array('text/x-comma-separated-values', 'text/comma-separated-values', 'application/octet-stream', 'application/vnd.ms-excel', 'application/x-csv', 'text/x-csv', 'text/csv', 'application/csv', 'application/excel', 'application/vnd.msexcel', 'text/plain');
if(!empty($_FILES['file']['name']) && in_array($_FILES['file']['type'],$csvMimes) ){
if(is_uploaded_file($_FILES['file']['tmp_name'])){
//open uploaded csv file with read only mode
$file_name = rand(1,9999).time().$_FILES['file']['name'];
move_uploaded_file($_FILES['file']['tmp_name'],"uploads/".$file_name);
///read file and seeds data in table
$sql = "LOAD DATA LOCAL INFILE 'uploads/$file_name'
INTO TABLE all_fee_data
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 6 LINES
";
$connection->query($sql);
}else{ // end checking if file uploaded or not
?>
<div class="alert alert-danger">Problem in uploading file! Plz check post_max_size and upload_max_filesize in php.ini once.</div>
<?php
}
} ///empty and mime check end
} /// isset check end
?>